Say that the benchmark page measures no joins at all - #402
Conversation
The conclusions generalise to "analytics". The evidence does not. All eight TSBS
queries are a scan, a filter and an aggregate over one relation:
$ grep -lci '\bjoin\b' queries/tsbs/*.sql | wc -l
0
$ grep -hiE '^\s*FROM' queries/tsbs/*.sql | sort | uniq -c
8 FROM %T
Star schemas and dimension joins are a large part of what columnar storage is
bought for, and this page has nothing to say about them. Read as written, it
invited a reader to carry a single-table result into a workload we have never
run.
A new section states the gap before the conclusions rather than after them, and
says plainly that the absence is unmeasured rather than bad. The two conclusions
that said "analytics" now say which analytics, and the closing line notes that
whether the result survives a join is open.
commandprompt#401 covers measuring it. This is the disclosure, which should not wait for the
measurement.
jdatcmd
left a comment
jdatcmd
left a comment
There was a problem hiding this comment.
Approved, and taking a follow-up
The placement is the part that makes this work: before the conclusions, not after.
A caveat under a conclusion is read by nobody who has already read the conclusion.
And the wording restraint is right: "Not that we do badly on them. We have not measured
them." Those are different claims and the page should not imply confidence in either
direction.
One thing it leaves ambiguous, which I can close with evidence
The section says nothing about whether joins work, only that their performance is
unmeasured. A reader can reasonably come away unsure whether a columnar table joins at
all. That is a worse impression than the truth, and I measured the truth earlier today
on a 200,000-row fixture:
| shape | result |
|---|---|
| columnar fact JOIN heap dimension, inner + aggregate | identical to heap/heap |
| LEFT JOIN with nulls | identical |
| heap on the outer side | identical |
| columnar JOIN columnar | identical |
semijoin (EXISTS) / anti-join (NOT EXISTS) |
identical |
| hash / merge / nested loop, each forced | all correct, all chosen |
Column projection also survives the join: the plan reports
Columnar Projected Columns: 1 of 4, so the columnar side reads only the join key
rather than losing the advantage at the join boundary.
So the accurate statement is stronger and more useful: joins are supported and
correct across access methods; their performance is unmeasured. I will add that in a
follow-up rather than send this back, since the change is additive and you have the
harder half in #401.
The one caveat worth carrying with it: foreign keys are not symmetric across the
boundary. A columnar table cannot be an FK parent, per test/fk_referencing.sh. That
belongs in limitations.md rather than the benchmark page, but it is the thing people
hit immediately after joins.
Merging.
Joins work; it is their speed that is unmeasured (#402 follow-up)
…401) Every other query on the benchmark page reads one table. Star schemas and dimension joins are a large part of what columnar storage is bought for, and we had no measurement of them. Not "we know and it is bad". We did not know. Five shapes: a no-join control, a selective dimension join, an unselective join, a multi-dimension star, and a wide projection under a join. Arms interleaved per shape rather than swept, because a sweep gives its first arm the cold cache (#271). TimescaleDB is supported and announces itself as skipped when absent. The result, at 20,000,000 rows on realistic data: no join, the control 0.53x we are 1.87 times FASTER selective dimension join 1.36x wide projection under a join 2.98x storage 0.141x 7.1 times smaller Read the first two together. Same rows, same bytes, same encoding. Our vectorized aggregate only sits directly above our scan, so a join between the scan and the aggregate disables it, and we then compete row at a time. A join does not cost us through the join. It costs us by disabling the thing we are fast at. The harness ASSERTS that mechanism instead of describing it. It fails the run if the control is not vectorized and fails it if the join arm is. Either half alone is consistent with the feature simply being switched off. Two mistakes of mine are built into its shape. The data shape is an arm, not an assumption, because my first result on this issue used random() float8 and reported a gap the realistic shape more than halves. And the no-join control is mandatory, because I once refuted my own correct hypothesis by running the decomposition with the vectorized aggregate turned off, which made both arms equally slow. docs/benchmarks.md carried "we have not measured them. See #401" from #402. That is now false, so it is replaced by the measurement rather than left to rot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqprqkCXuH8SegiZejE1Tw
2 participants
docs/benchmarks.mddraws conclusions about "analytics". Its evidence covers one shape ofanalytics.
The gap
All eight TSBS queries are a scan, a filter and an aggregate over one relation. The page
then closes with:
Star schemas and fact-to-dimension joins are a large part of what columnar storage is
bought for. Read as written, that sentence invites a reader to carry a single-table result
into a workload we have never run.
What changed
A What this page does not measure section, placed before the conclusions rather than
after them, saying which shapes are absent and that the absence is unmeasured rather
than bad. The two conclusions that said "analytics" now say which analytics, and the
closing line notes that whether the result survives a join is open.
Wording is deliberate on one point. It says we have not measured these shapes, not that we
are slow on them. We do not know either way, and the page should not imply confidence in
either direction.
Why now rather than with the measurement
#401 covers building a join-heavy fixture and measuring it. That is real work with a real
answer at the end. The disclosure is true today and costs nothing, and the page is linked
from the 1.0-alpha release notes.
Same class as #381 and #391: the page claiming more than the measurement behind it
supports. This one differs in that nothing measured is wrong, only the scope the
conclusions are drawn over.
Docs only.